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FOREWORD 


The  work  reported  here  was  conducted  over  a  period  of  a  little  more  than 
one  year  by  a  joint  team  of  Navy,  other  government,  industry,  and  academic 
experts  in  the  field  of  computer  operating  systems.  Only  a  few  of  the  Navy 
participants  were  actually  funded  to  directly  participate  in  this  process. 

The  report  was  funded  under  NUSC  Job  Order  Number  A45146,  Next 
Generation  Computer  Resources.  The  sponsoring  activity  is  Space  and  Naval 
Warfare  Systems  Command,  through  the  work  of  the  Operating  Systems  Standards 
Working  Group  (OSSWG) .  The  OSSWG  management  structure  is  as  follows: 

NGCR  Program  Manager,  Mr.  H.  Mendenhall,  SPAWAR  324 
NGCR  OSSWG  Co-Chairman,  CDR  R.  Barbour,  SPAWAR  324 
NGCR  OSSWG  Co-Chairman,  Ms.  T.  Oberndorf,  NADC 
Approach  Subgroup  Chairman,  Mr.  T.  Conrad,  NUSC 
Requirements  Subgroup  Chairman,  Mr.  R.  Bergman,  NOSC 
Available  Technology  Subgroup  Chairman,  Mr.  J.  Oblinger,  NUSC 

Although  the  report  is  the  result  of  work  performed  by  the  entire 
membership  of  the  OSSWG,  the  following  OSSWG  members  actively  performed  the 
evaluation  of  the  final  seven  candidates: 


CDR  Richard  Barbour 

SPAWAR  324 

Richard  Bergman 

NOSC 

Paul  Bickness 

Mitre 

Richard  Brogan 

Booz ,  Allen,  &  Hamilton 

Dale  Brouhard 

NOSC 

Gregory  Bussiere 

NUSC 

Antonio  Carangello 

Mitre 

Gordon  Caswell 

ESL 

Thomas  Conrad 

NUSC 

B.  Dasarathy 

Concurrent  Computer 

Larry  Daubert 

Rockwell  International 

Isobel  Davis 

Raytheon 

Steven  Davis 

DGM&S 

Dr.  Thomas  Drake 

Clemson  University 

Richard  Dvorchak 

Intel 

LT  Karl  Fairbanks 

NWC 

Gary  Fisher 

NIST 

Lester  Fraim 

Honeywell 

Dr.  Karen  Gordon 

IDA 

Dr.  Mars  Gralia 

JHU/APL 

Daniel  Green 

NAVSWC 

Raymond  Gretlein 

Dynamics  Research 

Joseph  Gwinn 

Raytheon 
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Barbara  Haleen 
James  Hall 
Neil  Henderson 
Gail  Holmes 
Steven  Howell 
John  Johnson 
Daniel  Juttelstad 
Kari  Kruempel 
Dr.  James  Leathrum 
Warren  Loper 
Dr.  Douglass  Locke 
Warren  Loper 
Michael  Morgan 

Dr.  John  F.  Nixon 

Patricia  Oberndorf 
James  Oblinger 
Frank  Prindle 
John  Reed 
Carl  Reinert 
Helmut  Roth 
Dr.  Timothy  Saponas 
John  Shea 
Del  Swanson 
Maria  Voreh 
Patrick  Watson 


Unisys 

NIST 

Litton  Data  Systems 

NUSC 
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Unisys 

Clemson  University 
Texas  Instruments 
IBM 
NOSC 

Pacific  International  Center  for 

High  Technology  Research  (PICHTR) 

General  Electric  Co.  Advanced 

Technology  Laboratories 
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IBM 
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CHAPTER  1 
INTRODUCTION 


The  Operating  Systems  Standards  Working  Group  (OSSWG)  evaluation  process 
defined  within  this  document  provides  the  techniques  and  methods  by  which  the 
OSSWG  will  recommend  a  baseline  operating  system  interface  specification  to 
the  Next  Generation  Computer  Resources  (NGCR)  program  office.  This  document 
describes  the  process,  the  inputs  into  the  process  in  the  form  of  criteria, 
baseline  candidates,  evaluators,  and  weights,  as  well  as  the  outputs  of  the 
process.  A  taxonomy  which  describes  the  relationships  and  organization  of  the 
criteria  is  presented  as  well  as  the  procedure  by  which  the  candidate 
baselines  are  to  be  evaluated. 


1 . 1  BACKGROUND 

The  U.S.  Navy  has  embarked  on  a  new  computing  resources  standardization 
effort  called  the  Next  Generation  Computer  Resources  (NGCR) .  This  program  is 
designed  to  fulfill  the  Navy's  need  for  standard  computing  resources  while 
allowing  it  to  take  advantage  of  commercial  products  and  investments  and  to 
field  new  technological  advances  more  quickly  and  effectively.  The  program 
revolves  around  the  selection  of  standards  in  10  interface  areas.  One  of 
these  is  an  operating  systems  interface  standard.  The  general  requirements 
for  this  interface  standard  are  that  it  be  Ada-oriented,  real-time, 
distributed/networked,  multi-level  secure,  reliable,  and  realizable  on 
heterogeneous  processors.  The  effort  to  establish  such  an  interface  standard 
was  initiated  at  the  start  of  1989  and  will  draw  on  industry  expertise.  An 
initial  operating  system  interface  standard  is  expected  in  1993  and  the  final 
standard  is  expected  to  be  usable  in  the  procurement  of  Navy  systems  in  fiscal 
year  1996. 

The  Navy's  current  computer  standardization  approach  is  having  difficulty 
remaining  competitive  in  an  environment  where  rapidly  changing  technologies 
permit  more  efficient  and  effective  solutions  to  the  range  of  Navy  computing 
system  requirements.  Thus,  the  objective  of  the  NGCR  program  is  to 
restructure  the  Navy's  approach  to  the  acquisition  of  standard  computer 
resources  so  as  to  take  better  advantage  of  commercial  advances  and 
investments.  It  is  expected  that  this  new  approach  will  result  in  reduced 
production  costs  (through  large  quantity  buys),  reduced  operation  and 
maintenance  costs,  avoidance  of  replication  of  Navy  RDT&E  costs  (for  separate 
projects  to  develop  similar  computers),  and  more  effective  system  integration. 

The  proposed  new  approach  is  an  open  systems  approach  based  on  the 
establishment  of  interface  standards  The  application  of  these  standards  will 


1-1 


NAVSWC  TR  90-248 


change  the  Navy's  approach  from  one  of  buying  standard  computers  to  one  of 
procuring  computer  resources  which  satisfy  the  interfaces  defined  by  the 
standards.  These  standards  will  be  applied  to  procurements  at  the  project 
level  rather  than  a  Navy-wide  procurement  level. 

These  interface  standards  will  be  based,  to  the  greatest  extent  possible, 
on  existing  standards.  In  cases  where  existing  industry  standards  do  not  meet 
Navy  mission-critical  needs,  the  approach  is  to  enhance  the  existing  standards 
jointly  with  industry,  thus  assuring  the  most  widely  accepted  set  of 
commercially-based  interface  standards  possible. 

An  operating  system  interface  standard  is  a  key  element  in  the  success  of 
NGCR.  The  function  of  the  operating  system  is  to  control  operation  of  all  the 
computing  system  hardware  and  software  elements  in  a  coordinated,  uniform 
manner  that  is  consistent  with  the  needs  of  embedded  and  real-time 
applications.  The  operating  system  capabilities  include  system 
initialization,  fault  tolerance  and  recovery,  global  resource  allocation,  and 
interprocess  communication.  The  operating  system  will  have  components  in  each 
processing  element.  The  operating  system  interface  standard  is  not  a  design 
of  the  operating  system  components  comprising  each  processing  system  but  is, 
in  part,  a  specification  of  an  application  program  interface  common  to  all 
computing  elements.  This  provides  the  basis  for  system-wide  dynamic  task  and 
resource  allocation.  Global  dynamic  task  and  resource  allocation  is  the  basis 
for  system-wide  fault  tolerance  and  recovery  in  heterogeneous  processing 
systems.  The  operating  system  provides  the  ability  to  achieve  multi-level 
security  at  the  system  level.  Conformance  to  other  Navy  directives  requires 
that  tne  operating  system  be  Ada-oriented. 

Other  ongoing  NGCR  standardization  efforts  are  in  the  backplane  and  local 
area  network  (SAFENET  I  and  SAFENET  II)  areas.  Each  of  these  efforts  has  a 
technically  oriented  working  group  comprised  of  industry,  academia,  and 
government  experts. 

The  OSSWG ,  the  operating  system  effort's  technical  working  group,  has 
been  tasked  to  evolve  an  interface  standard  for  operating  systems.  This 
document  defines  the  process  by  which  the  OSSWG  will  make  a  recommendation  of 
a  baseline  interface  specification  to  the  NGCR  program  office.  This  baseline 
specification  will  be  derived  from  one  or  more  existing  operating  system 
implementations,  specifications,  or  standards. 

The  OSSWG  is  organized  into  three  subgroups:  Available  Technologies, 
Approach,  and  Requirements.  The  Available  Technologies  Subgroup  is 
responsible  for  maintaining  an  extensive  knowledge  of  current  operating  system 
technologies.  The  Approach  Subgroup  is  responsible  for  defining  the 
evaluation  process.  In  addition,  it  is  also  responsible  for  defining  the 
programmatic  evaluation  criteria,  the  Representative  Application  Domains 
(RADs),  and  the  OSSWG  Reference  Model.  The  Requirements  Subgroup  is 
responsible  for  determining  and  delineating  Navy  requirements  for  operating 
systems.  Two  co-chairs,  Patricia  Oberndorf  and  CDR  Rick  Barbour,  lead  the 
OSSWG. 
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1.2  ORGANIZATION 

This  document  is  organized  into  5  chapters.  Chapter  2  defines  the 
approach  by  which  the  process  was  developed;  Chapter  3  defines  the  components 
of  the  evaluation  process  and  the  interrelationships  among  components;  Chapter 
4  defines  the  process  of  evaluating  operating  systems  specifications  against 
the  criteria;  and  Chapter  5  describes  the  types  of  conclusions  and  results 
that  are  expected  from  the  evaluation.  The  forms  to  perform  the  evaluation 
process  are  included  in  Appendix  A. 
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CHAPTER  2 

DEVELOPMENT  OF  EVALUATION  PROCESS 


The  evaluation  process  objective  is  to  allow  the  OSSWG  to  analyze  the 
current  state  of  operating  system  technology  in  order  to  make  recommendations 
of  a  baseline  interface  specification  to  the  NGCR  program  office.  The  OSSWG 
will  need  to  justify  its  recommendation;  therefore,  this  evaluation  process 
must  not  only  determine  the  'best'  candidate  interface  specif ication(s) ,  but 
also  describe  the  relative  strengths  and  weaknesses  of  each  of  the  possible 
candidates.  This  is  accomplished  by  organizing  the  criteria  into  a 
hierarchical  taxonomy  which  allows  analysis  of  results  at  either  a  high  gross 
level  or  at  a  low  detailed  level.  The  result  of  the  process  will  be  a  series 
of  values  related  to  the  relative  merits  of  the  candidate  interface 
specification  when  compared  against  key  attributes.  The  key  attributes  are 
service  classes  (operating  system's  functions),  programmatic  issues,  and 
representative  application  domains. 

Additionally,  the  process  attempts  to  be  as  objective  as  possible, 
demonstrating  neither  bias  toward  a  particular  candidate  baseline  nor  a  Navy 
application  domain.  For  instance,  a  small  number  of  biased  evaluators  scoring 
a  candidate  unusually  high/low  does  not  skew  the  results  of  the  process. 
Wherever  possible,  to  encourage  objectivity,  the  process  quantifies  the 
relative  differences  between  candidates. 

Figure  2-1  describes  the  inputs  and  the  outputs  of  the  evaluation 
process.  The  inputs  are:  the  criteria  which  are  the  basis  for  the  evaluation, 
the  candidate  baseline  specifications;  and  the  evaluators  of  the  candidates. 
The  output  of  the  process,  the  results,  will  be  a  recommendation  for  candidate 
baseline(s)  along  with  extensive  information  to  justify  the  recommendation.  A 
detailed  description  of  the  process  is  presented  in  Section  4.1. 


CRITERIA/REQUIREMENTS 
- ». 


CANDIDATES 

- ► 


EVALUATORS 


EVALUATION  PROCESS 


RESULTS 

- ► 


FIGURE  2-1.  EVALUATION  PROCESS  OVERVIEW 
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The  requirements  for  the  interface  have  been  developed  by  personnel  from 
industry,  academia,  the  Navy  as  well  as  other  government/DoD  agencies.  The 
requirements  have  been  derived  by  an  extensive  analysis  of  current  and  future 
Navy's  systems  needs.  Inputs  from  Navy  contractors,  various  Navy  program 
offices,  Navy  systems  documentation,  and  Navy  laboratory  personnel  were 
collected  to  generate  pertinent  requirements.  The  requirements  have  been 
arranged  into  two  categories:  Navy  functional  requirements  and  NGCR  program 
requirements.  The  Navy  functional  requirements  define  technical  needs  of  the 
operating  system  services  and  were  developed  by  the  Requirements  Subgroup  of 
the  OSSWG.  The  NGCR  program  requirements  define  criteria  related  to  the 
environment  in  which  the  NGCR  operating  system  standard  is  being  developed  and 
used.  NGCR  program  requirements  were  developed  by  the  Approach  Subgroup  of 
the  OSSWG.  An  example  of  a  functional  requirement  is:  the  types  of  process 
schedulers  that  must  be  supported.  An  example  of  an  NGCR  program  requirement 
is:  the  need  for  nonproprietary  interface  specifications.  The  Navy  functional 
requirements  are  listed  in  the  OSSWG  Requirements  Document  and  the  NGCR 
program  requirements  are  listed  in  Section  3.2. 

An  evaluation  criterion  is  the  atomic  unit  against  which  the  various 
candidates  are  evaluated,  producing  raw  scores.  Each  evaluation  criterion  is 
either  one  Navy  functional  and/or  NGCR  program  requirement,  with  an  associated 
metric  used  to  evaluate  candidate  interface  specifications.  The  evaluation 
criteria  is  applied  to  the  candidate  interface  specifications  which  are 
derived  from  existing  operating  systems  implementations,  designs  or  standards 
from  which  formalized  interfaces  can  be  extracted. 

Each  requirement  and  evaluation  criterion  is  defined  by  six  associated 
descriptors:  (1)  category  (criteria  number);  (2)  requirement/criteria  name; 

(3)  definition;  (4)  evaluation  criteria  metric;  (5)  rationale;  and  (6) 
reference/bibliography.  This  format  correlates  extremely  well  with  the  format 
which  the  Requirements  Subgroup  of  OSSWG  used  to  generate  Navy  functional 
requirements.  The  evaluation  criteria  have  been  derived  from  the 
requirements.  The  two  categories  for  requirements/criteria  are  either  Navy 
functional  or  NGCR  program.  Section  3.3  describes  the  relationship  between 
these  criteria  and  requirements. 

The  ultimate  NGCR  operating  system  interface  standard  is  intended  to 
adequately  meet  the  requirements  of  a  large  number  of  Navy  applications. 
Therefore,  the  evaluation  process  attempts  to  appraise  the  candidate  baselines 
against  the  full  breadth  of  Navy  operating  system  applications.  The  current 
breadth  of  applications  under  consideration  is  described  in  the  Reference 
Model  documentation  under  domain  areas.  Since  the  breadth  of  application 
domains  in  the  Navy  is  immense,  satisfactorily  representing  Navy  applications 
was  a  major  concern  in  the  development  of  the  evaluation  process.  It  was 
considered  impossible  to  evaluate  all  possible  candidate  interface 
specifications  against  the  criteria  as  they  relate  to  each  of  the  application 
domains.  Various  alternatives  were  considered  including  defining  one  most 
important  set  of  interfaces  and  services  (a  kernel),  and  defining  a  large 
number  of  representative  applications  such  as  the  cross  product  of  application 
domains  which  occur  in  the  NGCR  OSSWG  Reference  Model.  It  was  determined  that 
the  best  option  was  to  describe  a  small  number  of  application  types,  called 
the  Representative  Application  Domains  (RADs) ,  which  require  particular  system 
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service  and  interface  sets  and  are  truly  representative  of  operating  systems 
needs  of  applications  in  the  Navy.  This  set  will  satisfy  the  requirements  of 
a  wide  range  of  application  domains.  The  description  of  the  RADs  is  included 
in  Section  3.5. 
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CHAPTER  3 

EVALUATION  COMPONENT  DEFINITION 


This  chapter  describes  the  various  components  which  make  up  the 
evaluation  process  including  the  requirements,  criteria,  service  classes, 
programmatic  issues  and  the  RADs .  Section  3.1  and  3.2  reference  and/or  list 
the  requirements  for  the  standard;  Section  3.3  lists  the  evaluation  criteria; 
Section  3.4  describes  service  classes  and  programmatic  issues  including 
information  on  how  they  relate  to  the  criteria;  and  Section  3.5  defines  the 
RADs . 


Figure  3-1  shows  how  the  low  level  requirements  of  the  system  are 
aggregated  through  the  process  into  a  selection  of  the  baseline.  There  exists 
a  process  or  mapping  between  each  two  adjacent  levels.  The  mapping  between 
the  requirements  and  criteria  is  a  direct  one  to  one  mapping,  while  the 
mapping  from  criteria  to  services  classes,  programmatic  issues,  and  RADs  is  a 
many  to  one  mapping. 


SELECTION 


SERVICE  CLASSES 


RADs 


PROGRAMMATIC 

ISSUES 


QUANTIFIABLE  CRH 
functional 

[ERIA 

NGCR  program 

LOW  LEVEL  REQUIREh 
functional 

1ENTS 

NGCR  program 

FIGURE  3-1.  COMPONENT  RELATIONSHIP 


3.1  NAVY  FUNCTIONAL  REQUIREMENTS 

The  fu  ctional  requirements  used  in  the  evaluation  can  be  found  in  the 
Requirements  Document,  Version  2.0. 
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3.2  NGCR  PROGRAM  REQUIREMENTS 

The  following  is  the  set  of  NGCR  program  requirements.  The  category  of  all 
requirements  in  this  section  can  be  described  as  programmatic. 


3.2.1  Public  Domain  Interfaces 

3. 2. 1.1  Definition.  The  baseline  specification  must  be  in  the  public  domain 
and  should  not  be  proprietary. 

3.2. 1.2  Evaluation  Criteria. 

10- -currently  in  the  public  domain 

5- -proprietary ,  but  written  agreement  to  become  nonproprietary  if 
selected 

0- -proprietary 

3. 2. 1.3  Rationale .  The  standard  derived  from  the  baseline  specifications 
should  not  allow  one  company  or  group  of  companies  to  be  the  sole  supplier  of 
products  which  conform  to  the  standard.  This  open  approach  will  foster 
competition  between  commercial  vendors. 

3. 2. 1.4  Reference/Bibliographv.  NGCR  OSSWG  Charter,  Operational  Requirement 
for  Next  Generation  Computer  (pg  1) . 


3.2.2  Navy  Influence 

3. 2. 2.1  Definition.  This  is  the  extent  to  which  the  Navy  community  is  able 
to  influence  modification  and  adaptation  of  the  baseline  specification. 

3. 2. 2. 2  Evaluation  Criteria. 

10- -extensive  influence  with  voting  power 
5- -some  influence,  with  voting  power. 

0--no  influence 

3. 2. 2. 3  Rationale .  If  the  Navy  cannot  influence  the  baseline  specification, 
the  baseline  may  diverge  from  the  Navy  needs.  If  this  occurs  it  could  have 
one  of  two  effects.  One,  if  the  Navy  stays  with  the  baseline,  the  Navy  would 
have  a  standard  which  does  not  fit  the  Navy's  needs.  Two,  if  the  Navy 
proposes  a  new  baseline,  major  cost  and  schedule  problems  would  occur  in  the 
NGCR  operating  system  effort.  Therefore,  it  benefits  the  Navy  to  have  a  major 
influence  on  modifications  to  the  baseline  specification. 

3. 2. 2. 4  Reference/Bibl iographv 
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3.2.3  Maturity /Confidence 

3. 2. 3.1  Definition.  The  baseline  specification  should  be  mature  with  high 
confidence  so  that  the  baseline  specification  can  be  implemented  on  various 
hardware  configurations. 

3. 2. 3. 2  Evaluation  Criteria. 

10- -  implemented  and  used  over  time  by  a  wide  user  base 
5- -  implementations  have  just  become  available 
0--no  implementations  available 

3. 2. 3. 3  Rationale .  The  Navy  needs  to  use  a  standard  that  has  been  verified 
as  usable.  It  would  also  be  beneficial  to  the  government  if  implementations 
have  existed  for  sometime. 

3 . 2 . 3 . 4  Reference /Bibliography . 

3.2.4  Documentation 

3. 2. 4.1  Definition .  The  baseline  specification  must  be  well  documented. 

3.2. 4. 2  Evaluation  Criteria. 

10- -excellent ,  well  maintained  documentation  available 

5- -documentation  inadequate  for  operating  system  implementor 
and/or  user,  or  not  reflective  of  current  baseline 

0--no  documentation  exists 

3. 2. 4. 3  Rationale .  This  will  alleviate  problems  associated  with 
transitioning  the  baseline  specification  to  a  standard  and  problems  associated 
with  ambiguous  interfaces. 

3. 2. 4. 4  Reference/Bibliographv . 

3.2.5  Commercial  Acceptance  (Listed  as  Nonfunctional  on  Evaluation  Form) 

3. 2. 5.1  Definition .  The  baseline  specification  should  have  wide  commercial 
acceptance  or  show  great  promise  to  have  such  acceptance. 
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3. 2. 5. 2  Evaluation  Criteria. 

10- -accepted  by  a  wide  portion  of  the  commercial  community 

5- -partial  acceptance  from  a  portion  of  commercial  community 
exists  or  is  likely 

0--not  accepted  and  not  likely  to  be  accepted 

3. 2. 5. 3  Rationale .  A  wide  market  acceptance  will  encourage  implementations 
of  the  standard  to  be  implemented  without  Navy  funding. 

3. 2. 5. 4  Reference/Bibliographv .  Operational  Requirement  for  Next  Generation 
Computer  (pg  1),  NGCR  OSSWG  Charter. 


3.2.6  Timeframe 

3. 2. 6.1  Definition.  The  baseline  specification  should  be  ready  and 
available,  in  an  unified  form,  within  the  timeframe  of  the  OSSWG 
standardization  effort  without  Navy  funding. 

3. 2. 6. 2  Evaluation  Criteria. 

10- -available  now 

5- -not  available  now,  but  can  probably  be  available  by  1993 
0- -cannot  be  made  available  by  1993 

3.2.6. 3  Rationale .  This  will  reduce  risk  to  the  NGCR  program  while 
increasing  the  likelihood  of  commercial  acceptance.  The  initial  standard  is 
scheduled  to  be  in  place  by  January  1993. 

3. 2. 6. 4  Reference /Bibliography .  Operational  Requirements  for  Next  Generation 
Computer  (pg  1) 


3.2.7  User  Influence  (Listed  as  System/Standard  Goal  on  Evaluation  Form) 

3.2.7.1  Definition .  The  goal  of  the  development  of  the  baseline 
specification  should  have  been  to  meet  user's  needs. 

3.2. 7.2  Evaluation  Criteria. 

10- -Developed  as  industry  standard  with  wide  user  input 
5- -moderate  user  influence 
0--pure  research  with  no  user  influence 
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3. 2. 7. 3  Rationale .  The  NGCR  operating  system  interface  specification  is 
being  developed  to  meet  Navy  operating  system  user  needs.  A  baseline 
developed  with  the  same  motivation  as  NGCR  is  more  likely  to  be  applicable  to 
this  program. 

3. 2. 7. 4  Reference/Bibliographv . 

3.2.8  Economics /Cost 

3. 2. 8.1  Definition .  Implementing  the  operating  systems  which  conform  to  the 
baseline  specification  should  be  achievable  by  vendors  from  an  economic 
viewpoint . 

3. 2. 8. 2  Evaluation  Criteria. 

10- -low  cost  to  implement  will  result  in  implementations  by  many  vendors 

5- -moderate  cost  to  implement  will  result  in  some  Navy  unique 
implementations 

0- -extreme  cost  to  implement  will  result  in  predominately  Navy 
implementations 

3. 2. 8. 3  Rationale .  A  baseline  specification  which  is  expensive  to  implement 
will  be  expensive  to  the  Navy.  It  will  likely  have  very  few  implementations 
and  will  not  gain  broad  commercial  acceptance.  The  Operational  Requirement 
Document  states  that  existing  software  environments  have  limited  productivity. 

3. 2. 8. 4  Reference/Bibliographv .  Operational  Requirement  for  Next  Generation 
Computer  (pg  1). 

3.3.  EVALUATION  CRITERIA 

There  is  a  one  to  one  mapping  between  criteria  and  requirements; 
therefore  each  requirement  referenced  in  3.1  and  listed  in  3.2  will  be  used  as 
the  evaluation  criteria. 


3.4  SERVICE  CLASSES  AND  PROGRAMMATIC  ISSUES 


This  section  describes  services  classes  and  programmatic  issues. 
Included  in  the  description  is  an  explanation  of  how  the  various  service 
and  programmatic  issues  are  related  to  the  criteria  of  3.3. 


class 
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3.4.1  Service  Classes 

The  service  classes  for  the  evaluation  are  listed  in  the  Requirements 
Document.  A  detailed  explanation  of  each  service  class  is  given  in  the  OSSWG 
Reference  Model.  The  service  classes  used  in  the  evaluation  are: 

1.  General  Requirements 

2.  Architecture -Dependent  Services 

3.  Capability  and  Security  Services 

4.  Data  Interchange  Services 

5.  Event  and  Error  Management  Services 

6.  File  Services 

7.  Generalized  I/O  Services 

8.  Network  and  Communications  Services 

9.  Process  Management  Services 

10.  Project  Support  Environment  Interaction  Services 

11.  Reliability,  Adaptability,  and  Maintainability 

12.  Resource  Management  Services 

13.  Synchronization  and  Scheduling  Services 

14.  System  Initialization  and  Reinitialization  Services 

15.  Time  Services 

16.  Ada  Language  Support  Services 


3.4.2  Programmatic  Issues 

There  is  a  one  to  one  mapping  between  NGCR  program  requirements  and 
programmatic  issues. 


3.5  REPRESENTATIVE  APPLICATION  DOMAIN  SET 

The  RADs  define  characteristic  application  types  which  provide  coverage 
for  a  wide  range  of  Navy  applications.  Each  application  in  the  set  is 
described  by  its  relative  inclusion/exclusion  of  key  attributes  described  in 
the  Reference  Model.  The  representative  applications  are  defined  in  the 
following  sections. 

For  each  RAD,  weights  are  attached  to  service  classes  allowing  certain 
service  classes  to  be  worth  more  than  others,  depending  on  how  they  relate  to 
the  representative  application  domain.  Service  classes  have  as  many  weights 
as  representative  applications,  with  each  application's  weight  corresponding 
to  the  relative  importance  of  that  service  class  to  that  application 
requirement.  A  more  detailed  description  of  the  weighting  process  is  included 
in  Chapter  4.  Appendix  B  includes  the  service  class  and  the  class'  weight  for 
each  application  domain. 
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3.5.1  Application  Domain  Ruby 

This  application  domain  frequently  features  on-line  transaction 
processing,  off-the-shelf  software  products,  networking  to  PC’s,  workstations, 
other  host  environments,  and  background  processing.  This  application  domain 
is  characterized  by  strong  requirements  for  data  management,  data 
reformatting,  file  services,  generalized  I/O,  and  resource  management.  By 
contrast,  specific  requirements  for  operating  system  support  for  languages  and 
an  interface  to  the  project  support  environments  are  low.  This  domain 
includes  shore-based  logistics  systems,  for  example.  In  extant  technology, 
implementations  of  these  systems  typically  involve  wide  area  networks  of 
multiple  heterogeneous  processors  linked  through  gateways  and  the  like. 


3.5.2  Application  Domain  Opal 

This  application  domain  consists  of  special  purpose  dedicated 
processors,  high  data  rates,  and  computationally  intensive  cyclic  processing. 
This  application  domain  is  characterized  by  strong  requirements  for  event  and 
error  management,  generalized  I/O  and  times  services.  By  contrast,  specific 
requirements  for  operating  system  support  for  data  management,  file  system, 
and  man-machine  interfaces  are  minimal.  In  extant  technology,  implementations 
of  these  systems  typically  involve  one  or  more  specialized  processors  such  as 
iright  be  found  in  signal  processing  applications. 


3-5.3  Application  Domain  Amethyst 

This  application  domain  consists  of  message  switching,  store  and  forward, 
message  processing  encryption,  and  error  detection  and  recovery.  This 
application  domain  is  characterized  by  strong  requirements  for  security, 
fault- tolerance ,  nuclear  survivability,  event  and  error  management,  networking 
and  communications,  scheduling  and  synchronization,  and  time  management.  By 
contrast,  specific  requirements  for  operating  system  support  for  project 
support  environments  are  low.  In  extant  technology,  implementations  of  these 
systems  involve  processors  on  a  given  platform  that  must  interface  with 
networks  that  are  widely  distributed  or  with  intra-platform  networks.  For 
example,  this  would  include  processors  interfacing  with  global  command, 
control  and/or  intelligence  systems. 


3.5.4  Application  Domain  Garnet 

This  application  domain  consists  of  autonomous  embedded  processors  with  a 
wide  spectrum  of  data  rates  and  duty  cycles.  Overall,  this  application  domain 
does  not  put  high  demands  on  the  operating  system.  This  application  domain  is 
characterized  by  strong  requirements  for  language  support,  reliability  and 
availability.  By  contrast,  specific  requirements  for  operating  system  support 
for  security,  data  management,  file  services,  man-machine  interfaces,  and 
network  and  file  communications  are  low.  This  application  domain  is 
exemplified  by  single  processors  embedded  in,  for  example,  missile  warheads, 
torpedoes,  and  shipboard  guns. 
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3.5.5  Application  Domain  Topaz 

This  application  domain  is  characterized  by  high  computational  needs, 
interface  to  multiple  sensors,  and/or  controls,  and  support  of  interactive 
displays.  Overall,  this  application  domain  puts  high  demands  on  the  operating 
system.  This  application  domain  has  strong  requirements  for  operating  system 
support  for  languages,  data  management,  data  reformatting,  man-machine 
interfaces,  and  reliability  and  availability.  These  applications  include 
major  subsystems  of  shipboard  systems  such  as  navigation  systems,  ship 
control  systems  or  command  systems.  In  extant  technology,  such  applications 
typically  include  heterogeneous  processors  directly  communicating  on  a  near- 
continuous  basis.  One  processor  is  typically  a  high-speed  graphics  processor. 


3.5.6  Application  Domain  Emerald 

This  application  domain  consists  of  mission  critical  systems  which  are 
characterized  by  nuclear  safety,  command  significance,  and  large 
ramifications  of  system  failure.  This  application  domain  has  strong 
requirements  for  operating  system  support  for  security,  reliability  and 
availability.  By  contrast,  specific  requirements  for  operating  system  support 
for  files  services  are  low.  In  extant  technology  these  applications  are 
frequently  embodied  in  single  processors  exercising  centralized  control  over 
other  processors  and/or  devices.  The  application  includes,  for  example, 
processors  that  control  the  enabling,  targeting,  and  firing  of  strategic 
weapons.  These  systems  typically  include  requirements  for  access  control  and 
man-machine  interface  management. 


3.5.7  Application  Domain  Diamond 

This  application  domain  consists  of  networked  dedicated  processors 
connected  to  multiple  sensors,  controls,  and  displays.  This  application 
domain  is  characterized  by  strong  requirements  for  operating  systems  support 
for  languages,  hardware  architecture  dependencies,  event  and  error 
management,  reliability  and  availability,  scheduling  and  synchronization,  and 
time  services.  By  contrast,  specific  requirements  for  operating  system 
support  for  data  management  and  file  services  are  low.  This  application 
domain  typically  includes  multiple  heterogeneous  processors  with 
particularized,  dedicated  functions.  These  processors  are  linked  for 
cooperative  interaction  as  might  be  found,  for  example,  in  avionics 
app 1 icat ions . 


3.5.8  Application  Domain  Sapphire 

This  application  domain  consists  of  many  cooperating  subsystems  that 
carry  out  mission  critical  functionality.  Overall,  this  application  domain 
puts  high  demands  on  the  operating  system.  This  application  domain  is 
characterized  by  strong  requirements  for  operating  system  support  for 
languages,  security,  networking  and  communication,  process  management,  project 
support  environment,  reliability  and  maintainability,  and  time  services.  By 


3-8 


NAVSWC  TR  90-248 


contrast,  specific  requirements  for  operating  system  support  for  file  services 
are  low.  This  application  domain  typically  includes  multiple  platform- local 
networks  of  large  numbers  of  heterogeneous  processors.  Examples  include  large 
tactical  combat  systems  such  as  might  be  found  aboard  major  surface  ships  and 
submarines . 
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CHAPTER  4 

EVALUATION  PROCESS  APPLICATION 


Figure  4-1  describes  the  overall  evaluation  process.  The  inputs  to  the 
evaluation  process  are  the  criteria  which  are  the  basis  for  the  evaluation, 
the  candidate  baseline  specifications,  and  the  evaluators  of  the  candidates. 
The  output  of  the  process,  the  results,  will  be  a  recommendation  for  candidate 
baseline(s)  along  with  extensive  information  to  justify  the  recommendation. 

The  total  number  of  possible  baseline  candidates  is  reduced  by  an  informal 
pre-screening  process  by  the  Available  Technology  Subgroup.  The  product  of 
this  early  screening  process  is  the  baseline  candidates.  The  baseline 
candidates  along  with  the  relevant  requirements  and  criteria  developed  by  the 
OSSWG  are  input  into  the  scoring  process.  In  the  scoring  process,  the 
functional  and  programmatic  criteria  are  used  to  evaluate  each  of  the  baseline 
candidates.  In  addition  to  a  raw  score,  the  evaluators  may  also  enter  their 
confidence  level  (rated  low,  medium,  or  high)  and  rationales/comments. 

The  outputs  from  the  scoring  process  are  the  raw  scores.  These  raw 
scores  are  then  refined  and  the  number  of  scores  reduced  by  the  use  of  a 
filtering  process.  The  raw  scores  are  reduced  from  three  to  two  dimensions. 
The  three  dimensions  of  raw  scores  are  the  baseline  candidate,  criterion,  and 
evaluator.  In  contrast,  the  filtered  scores,  called  Criterion  Scores,  have 
two  dimensions:  baseline  candidate  and  criterion.  A  full  description  of  the 
filtering  process  is  given  in  Section  4.1.4. 

The  scores  are  then  further  processed  by  weighting  the  importance  of  a 
Criterion  Score  to  a  particular  service  class  for  most  service  classes.  The 
results  of  applying  these  weights  is  a  value  for  each  candidate  interface 
specification  against  each  service  class  or  programmatic  issue.  These  values 
are  the  processed  scores  that  are  used  in  the  analysis. 

The  final  set  of  processed  scores  is  derived  by  applying  weights  to  the 
service  class  scores.  These  representative  application  weights  describe  the 
relative  importance  of  particular  service  classes  to  particular  representative 
applications.  The  result  of  the  employment  of  these  weights  to  the  scores  is 
a  set  of  values  which  will  show  how  well  the  various  candidate  interface 
specifications  satisfy  the  requirements  of  representative  Navy  applications. 

Various  meetings,  approximately  six  weeks  apart,  are  held  by  the  OSSWG 
during  the  evaluation  process.  During  the  evaluation  process  the  following 
meetings  will  occur  (listed  in  time  order) :  Process  Preparation  Meeting(s) , 
Baseline  Candidates  Introduction  Meeting,  Detailed  Candidate  Evaluation 
Meeting,  and  Process  Results  Meeting(s). 
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The  remaining  portion  of  this  chapter  describes  in  more  detail  the  steps 
in  the  evaluation  process. 
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FIGURE  4-1.  EVALUATION  PROCESS 


4-2 


NAVSWC  TR  90-248 


4.1  SCORING  STEPS 


4.1.1  Early  Screening  Process 

The  Early  Screening  Process  is  the  mechanism  by  which  the  total  number 
of  possible  baseline  candidates  is  reduced  by  an  informal  pre-screening 
process  by  the  Available  Technology  Subgroup  of  the  OSSWG .  This  reduces  the 
number  of  possible  operating  system  specifications  to  the  number  of  candidate 
interface  specif ications  which  are  truly  viable.  This  is  done  by  informally 
applying  two  methods  described  in  this  section  and  eliminating  obviously 
inadequate  candidates.  By  the  Baseline  Candidates  Introduction  Meeting,  the 
baseline  candidates  are  selected  and  a  brief  description  of  each  candidate  is 
presented . 

The  two  methods  used  to  perform  the  early  screening  process  are  the 
decision  option  paper  method  (DOP  method)  and  the  positive-negative  method  (PN 
method) .  The  first  method  is  based  on  a  comparison  of  operating  system 
capabilities  against  DOP  technology  area  requirements.  For  the  DOP  method  six 
critical  criterion  area  were  defined:  (1)  support  of  real-time,  (2) 
distributed,  (3)  fault  tolerance,  (4)  security,  (5)  Ada  language,  and  (6) 
heterogeneous  processors.  This  scoring  system  is  a  set  of  marks  consisting  of 
("+"  provides  some  support,  provides  no  support,  "?"  insufficient 

information,  or  "blank"  no  support  but  can  easily  be  added). 

In  the  PN  method  two  sets  of  criteria  are  defined,  one  for  selecting 
candidates  from  the  original  list  which  should  be  included  on  the  candidate 
list,  and  the  other  for  determining  operating  systems  which  were  unlikely  to 
be  usable  as  a  primary  operating  system  interface  standard. 

A  matrix  using  the  two  methods  is  developed  from  the  results  of  the  two 
methods.  The  matrix  illustrates  where  the  two  methods  tend  to  support  each 
other.  In  instances  where  the  results  appear  to  conflict,  attempts  are  made 
to  review  the  individual  results  and  adjust  if  appropriate,  or  in  most  cases 
to  provide  an  explanation. 


4.1.2  Weighting 

The  scores  are  weighted  in  respect  to  their  relative  importance  to  the 
key  attributes  of  the  process.  There  are  two  sets  of  weights.  The  first  set 
of  weights,  Weight  Set  1,  describes  the  relative  importance  of  a  Criterion 
Score  to  a  particular  service  class.  The  results  of  applying  these  weights  is 
a  value  for  each  candidate  interface  specification  against  each  applicable 
service  class.  The  second  set  of  weights,  Weight  Set  2,  describes  the 
relative  importance  of  service  classes  and  their  scores  to  representative 
applications.  The  result  of  the  employment,  the  second  set  of  weights  to  the 
scores,  is  a  set  of  values  which  shows  how  well  the  various  candidate 
interface  specifications  satisfy  the  requirements  of  representative  Navy 
app 1 i cat  ions . 
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For  Weight  Set  1,  it  was  determined  that  since  the  criteria  of  service 
class  1  (General)  are  not  necessarily  related  to  each  other,  no  weights  should 
be  generated  for  this  service  class.  Additionally,  for  programmatic  criteria, 
it  was  decided  that  the  NGCR  program  office  would  be  the  best  organization  to 
determine  the  relative  importance  of  criteria  within  this  class. 

Weight  Set  1  is  developed  by  the  OSSWG  committee  at  the  Baseline 
Candidates  Introduction  Meeting.  The  weights  for  Weight  Set  1  are  generated 
by  full  OSSWG  membership.  Members  submit  their  set  of  raw  weights  on  a  form 
similar  to  the  evaluation  form  for  each  of  the  steps  in  the  process  that 
requires  weights.  The  submission  is  collected  by  the  co-chairs.  The 
filtering  process  described  in  Section  4.1.4  is  applied  to  the  candidate 
weight  assignments.  The  final  weighting  decisions  are  made  by  the  OSSWG  co¬ 
chairs  and  Weight  Set  1  is  not  disclosed  to  the  OSSWG  general  membership  until 
after  the  scoring  process  of  criteria  against  the  viable  candidates  is 
complete . 

Weight  Set  2  is  developed  during  Process  Preparation  Meetings  by  the 
Approach  Subgroup  of  OSSWG  and  is  set  by  the  Baseline  Candidates  Introduction 
Meeting. 

By  the  beginning  of  the  scoring  process,  both  weight  sets  are  fixed. 


4.1.3  Scoring  Process 

The  scoring  process  involves  qualified  evaluators  rating  each  baseline 
candidate  against  various  criteria.  Each  qualified  evaluator  signs  up  for  the 
criterion  areas  (service  classes  and/or  programmatic  issues)  which  the  member 
feels  qualified  to  evaluate.  Given  the  limited  amount  of  time  and  effort  a 
member  is  able  to  dedicate  to  the  evaluation,  he/she  should  restrict  the 
number  of  criterion  areas  that  he/she  evaluates.  The  co-chairs  adjust  the 
assignment  so  that  a  sufficient  number  (at  least  seven)  of  evaluators  score 
each  criterion. 

The  evaluators  are  required  to  score  all  candidate  baselines  against  a 
particular  criterion  and  further,  are  required  to  evaluate  all  criterion 
within  a  service  classes/programmatic  issues  they  are  evaluating.  The 
filtering  process  will  discard  scores  of  an  evaluator  for  a  service  class  when 
the  evaluator  did  not  provide  all  necessary  scores. 

A  trial  run  of  the  scoring  process,  after  the  Baseline  Candidates 
Introduction  Meeting,  is  performed  to  finalize  the  scoring  process.  This 
gives  the  chance  for  the  evaluation  process  to  be  tested  and  modified,  if 
needed . 
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The  qualified  evaluators  receive  the  evaluation  package  after  the  trial 
scoring  process  is  complete,  but  before  the  Detailed  Candidate  Evaluation 
Meeting.  This  package  includes  documentation  of  each  of  the  candidate 
baselines,  this  document,  the  criteria  with  supporting  requirements,  the 
reference  model,  sample  scoring  form,  and  scaring  instructions.  An  advocate 
for  each  baseline  candidate  will  make  a  presentation  to  the  evaluators  at  the 
Detailed  Candidate  Evaluation  Meeting.  During  the  remainder  of  the  Detailed 
Candidate  Evaluation  Meeting  and  during  the  next  three  weeks  the  evaluators 
will  have  an  opportunity  to  score  each  of  the  candidate  baselines  against  each 
of  the  criterion.  The  evaluators  have  two  options  for  transmitting  their 
scores  to  NADC ,  where  the  raw  score  processing  occurs.  The  deadline  for  the 
evaluator  to  send  scores  is  set.  The  deadline  for  sending  a  hardcopy  listing 
of  scores  is  one  week  earlier  than  the  deadline  for  scores  transmitted  over 
DDN  (or  the  automated  evaluation  form) .  This  is  due  to  the  logistics  problems 
in  handling  hardcopy  scores  (entry  and  verification  process) .  This  deadline 
can  be  modified  at  the  co-chairs  discretion. 

By  five  working  days  before  the  beginning  of  the  Process  Results  Meeting 
all  scoring  is  given  the  OSSWG  co-chairs.  The  raw  scores  are  filtered  by  the 
methods  described  in  Section  4.1.4.  The  results  are  tabulated  by  the  process 
described  in  Section  4.4  The  results  of  the  processing  of  the  raw  scores  into 
the  various  processed  scores  are  presented  to  the  OSSWG  at  the  Process  Results 
Meeting(s) . 


4.1.4  Raw  Score/Weight  Filtering  Process 

A  filtering  process  is  applied  to  various  scores  for  many  reasons: 
elimination  of  scoring  anomalies,  to  account  for  the  variable  number  of  raw 
scores/weights  for  each  attribute,  and  to  generate  an  overall  score  for  a  key 
attribute.  Filters  are  applied  to  the  raw  criterion  scores  to  generate  the 
Criterion  Score  for  each  criterion  and  to  the  raw  weights  to  develop  the 
Service  Class  Weights  and  Representative  Application  Domain  Weights. 

4. 1.4.1  Criterion  Filtering  Process.  This  section  describes  the  filtering 
process  to  determine  the  Criterion  Score  from  the  raw  scores  for  each 
criterion  and  candidate  baseline.  The  scoring  of  each  criterion  for  each 
candidate  baseline  is  performed  by  evaluators  who  are  qualified  through  the 
process  described  in  Section  4.2.1.  The  number  of  raw  scores  per 
criterion/candidate  baseline  must  be  at  least  seven.  First,  the  filtering 
process  discards  scores  of  an  evaluator  of  a  particular  criterion  if  the 
evaluator  did  not  score  the  criterion  on  ail  baseline  candidates.  The  mean  of 
the  remaining  scores  for  each  criterion/candidate  baseline  pair  is  the 
Criterion  Score  for  the  pair. 

4. 1.4.2  Service  Class  Weight  Filtering  Process.  This  section  describes  the 
filtering  process  to  determine  the  weights  on  the  services  classes  used  to 
generate  scores  for  each  criterion.  The  raw  weights  are  generated  by  the  OSSWG 
membership.  The  number  of  raw  weight  inputs  per  criterion  must  be  at  least 
seven.  The  process  to  filter  these  weights  to  arrive  at  the  one  filtered 
weight  is  the  mean  of  the  weights. 
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4. 1.4. 3  RAD  Weight  Filtering  Process.  This  section  describes  the  filtering 
process  to  determine  the  weights  on  the  services  classes  used  to  generate 
scores  for  each  RAD.  The  raw  weights  for  the  RAD  scoring  process  are 
generated  by  the  Approac.  '  Subgroup  of  the  OSSWG.  The  number  of  raw  weight 
inputs  per  service  class,  RAD  pair  must  be  at  least  seven.  The  process  to 
filter  these  weights  to  arrive  at  the  one  filtered  weight  is  the  mean  of  the 
weights . 


4.2  THE  EVALUATORS 

This  section  describes  the  qualification  of  evaluators  as  well  as  the 
responsibilities  of  each  evaluator.  The  qualification  process  is  the 
procedure  by  which  evaluators  are  selected  and  committed  to  the  candidate 
scoring . 


4.2.1  Evaluator  Qualification 

Organizations  outside  the  Navy  which  have  sent  representatives  to  attend 
at  least  two  OSSWG  meetings  by  the  Baseline  Candidates  Introduction  Meeting 
are  permitted  to  qualify  organizational  personnel  to  the  candidate  scoring. 

If  such  an  organization  has  sent  two  or  more  representatives  to  two  or  more 
meetings  then  the  organization  will  be  permitted  to  qualify  two  evaluators  for 
the  candidate  scoring;  otherwise,  it  will  be  permitted  to  qualify  one 
evaluator.  Organizations  within  the  Navy  are  not  limited  in  the  number  of 
representatives  they  qualify.  An  organization  is  defined  as  a  corporate 
division,  company  or  college.  Final  determination  of  separate  organizational 
units  will  be  made  by  the  OSSWG  co-chairs. 

4.2.2  Evaluator  Responsibilities 

This  process  calls  for  a  larg  '  T  n-t  aM  offort  commitment  by  each 
evaluator.  This  fact  is  communicated  to  all  evaluators  qualified.  Evaluators 
should  be  of  the  highest  technical  expertise  in  the  area  of  operating  systems. 
The  qualification  process  allows  a  company  to  substitute  more  experienced 
technical  personnel  for  the  personnel  regularly  attending  OSSWG  meetings. 

This  encourages  qualified  marketers  or  managerial  personnel  to  substitute 
highest  technical  personnel  for  themselves. 

Each  evaluator  qualified  by  an  organization  is  required  to  send  a  letter 
of  intent  to  participate  in  the  process  to  one  of  the  OSSWG' s  co-chairs  by  a 
set  duration  (approximately  one  week)  after  the  Baseline  Candidates 
Introduction  Meeting  and  before  the  evaluation  packages  are  sent. 

The  evaluator  is  responsible  for  attending  the  Detailed  Candidate 
Evaluation  Meeting,  if  at  all  possible.  The  evaluator  is  also  responsible  for 
returning  all  evaluation  forms,  completed,  by  the  timetable  described  in 
Section  4.1.3. 
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4.3  EVALUATOR  GUIDELINES 

The  following  section  describes  the  guidelines  and  procedures  by  which 
the  weight  and  score  developer  generates  the  scores.  Rules  of  evidence  are 
addressed  along  with  a  description  relative  to  scoring/weighting  and  the 
process  of  filling  out  the  scoring  forms. 


4.3.1  Weighting  Guidelines 

All  weights  are  on  a  zero  to  ten  scale  with  zero  being  the  lowest; 
possible  score  and  ten  being  the  highest.  Due  to  the  mathematical  process  of 
combining  weights  and  scoring,  the  ratio  of  weights  relative  to  each  other  is 
the  important  factor  when  comparing  weights  for  a  particular  final  score.  In 
other  words,  an  attribute  with  the  weight  of  two  is  twice  as  important  as 
another  with  a  weight  of  one,  just  as  an  attribute  with  the  weight  of  ten  is 
twice  as  important  as  an  attribute  with  the  weight  of  five. 

To  arrive  at  the  weights,  which  describe  the  relative  worth  of  scores  in 
the  process,  the  weighter  determines  how  important  the  attribute  to  be 
weighted  is  to  the  characteristic  to  be  described  by  the  outcome  of  the 
weighted  score.  The  zero  score  means  there  is  no  relationship  between  the 
attribute  and  the  characteristic.  A  score  of  five  means  there  is  a  moderate 
relationship,  and  a  score  of  ten  means  there  is  a  critical/essential 
relationship . 


4.3.2  Scoring  Guidelines 

All  scores  are  on  a  zero  to  ten  scale  with  zero  being  the  lowest 
possible  score  and  ten  being  the  highest.  The  description  evaluation  metric 
for  the  zero,  five  and  ten  score  is  included  with  each  criterion.  The 
evaluator  should  score  each  baseline  candidate  against  each  criterion 
separately,  judging  strictly  on  how  well  the  baseline  candidate  meets  the 
criterion  and  scoring  given  the  evaluation  metric. 

Unless  overwise  specified  in  the  criteria,  the  evaluator  rates  the 
candidate's  current  interface  capabilities  against  the  criteria. 

4. 3. 2.1  Using  Evaluation  Forms.  Each  qualified  evaluator  receives  an 
evaluator  identification  number.  This  evaluator  identification  number  is 
entered  into  each  of  the  evaluator's  evaluation  forms.  Evaluator 
identification  numbers  are  used  to  keep  the  evaluator  scores  anonymous,  but 
allows  accountability  for  all  scoring  forms.  The  OSSWG  co-chairs  will  know 
each  evaluator's  identification  number  to  ensure  no  duplication  or  erroneous 
identification  numbers. 

The  scores  are  entered  by  each  qualified  evaluator  into  the  evaluation 
forms.  Each  form  is  used  to  evaluate  a  particular  candidate  baseline  against 
the  criteria  associated  with  particular  service  classes.  The  evaluator  must 
enter  into  each  form  his/her  evaluator  identification  number.  In  addition, 
each  candidate  baseline  has  an  identification  code  which  the  evaluator  must 
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also  enter.  For  each  criterion  the  evaluators  enter  their  raw  score,  from 
zero  to  ten  as  described  in  Section  4.3.2.,  on  the  form.  Optionally,  the 
evaluators  enter  their  confidence  in  the  raw  score  accuracy,  either  high, 
medium  or  low.  If  a  confidence  level  is  not  entered,  it  is  assumed  to  be 
medium.  Comments  and  rationale  concerning  their  score  including  its  rationale 
or  reference  to  information  used  to  arrived  at  the  score  is  solicited,  but  is 
not  mandatory.  Appendix  A  is  the  template  for  the  evaluation  forms. 

4. 3. 2. 2  Rules  of  Evidence .  This  section  describes  the  information  that  an 
evaluator  can  use  to  arrive  at  the  raw  criteria  score  and  confidence  level. 

The  evaluators  are  given  a  packet  of  information  from  the  OSSWG  to  perform  the 
evaluation.  This  documentation  along  with  the  briefing  given  at  the  Detailed 
Candidate  Evaluation  Meeting  provides  the  major  source  of  information. 

If  an  evaluator  is  aware  of  additional  information  which  is 
documentable ,  then  the  evaluator  can  use  this  information  in  the  scoring  of 
the  criteria.  However,  the  evaluator  must  specify  in  the  rationale  section  of 
the  evaluation  form  the  additional  source  of  information.  Sources  of 
information  that  should  be  avoided  include  informal  conversations,  whether 
they  be  with  marketeers  or  technical  personnel. 


4.4  SCORE  PROCESSING 

This  section  describes  the  method  by  which  the  raw  scores  for  each 
candidate  baseline  specification  are  processed  in  order  to  arrive  at  the 
processed  scores. 

The  processing  of  the  scores  is  as  automated  as  possible.  The  programs 
to  perform  data  reduction  through  the  filtering  and  applying  weights  are 
implemented  on  a  computer.  Additionally,  evaluation  forms  sent  via  DDN  are 
automatically  read  by  a  computer,  allowing  it  to  automatically  find  erroneous 
or  missing  data. 

For  each  candidate  baseline  specification,  the  final  score  for  each 
criterion,  the  Criterion  Score,  is  arrived  at  by  the  filtering  method 
described  in  Section  4.1.4.  The  filtering  process  not  only  reduces  the  effect 
of  bias  and/or  numeric  aberrations  but  also  reduces  the  dimensions  of  the 
scoring  values  from  three  to  two  by  eliminating  the  evaluator  dimension. 

The  Service  Class  Score  for  each  candidate  baseline  specification  and 
service  class  is  determined  by  multiplying  the  Criterion  Scores  for  the 
candidate  baseline  specification  by  the  respective  weight  for  the  service 
class.  The  Programmatic  Issue  Score  for  each  candidate  baseline  specification 
and  programmatic  issue  is  determined  by  multiplying  the  Criterion  Score  by  the 
respective  weight  for  the  programmatic  issue.  The  Representative  Application 
Score  for  a  particular  application  and  candidate  baseline  specification  is 
defined  as  the  summation  of  multiplying  each  representative  application's 
weight  for  each  service  class  by  the  corresponding  Service  Class  Score  of  the 
candidate  baseline  specification. 
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The  mathematical  representation  of  scores  for  each  baseline  candidate 
follows : 


Sy  ->  is  the  raw  score  for  criterion  i  from  evaluator  j 

Si  ->  is  the  Criterion  Score  for  criterion  i 

Si  -  fA  ( S ij )  where  fA  is  the  mean  of  raw  scores  for 
criterion  i 

tflk  ->  is  the  weight  for  score  Si  for  service 
class/programmatic  issue  k 

Ck  ->  is  the  set  of  criterion  which  belong  to  service 
class/programmatic  issue  k 

Sk  ->  is  the  normalized  Criterion  Score  for  service 
class/programmatic  issue  k 

Sk  -  E(  Si  Wlk  /  2(  Wjk  )  )  for  each  service  class/programmatic 
i  c  Ck  j  f  Ck  issue  k 

->  is  the  weight  on  service  class  k  for  representative 
application  m 

Atn  ->  is  the  set  of  service  classes  which  belong  to 
representative  application  m 

R,,,  ->  is  the  Representative  Application  Score  for  domain  m 

Rjd  ”  ^  Wkm  Sk 

k  «  An, 

Weights  are  arrived  at  through  a  filtering  process  so  that: 

Wy  -  fw  (wijk)  where  fw  function  is  the  mean  for  evaluator  i, 
for  attributes  j  and  k. 

The  main  results  of  the  evaluation  that  is  used  in  the  analysis  and 
recommendation  are  the  Representative  Application  Score  (Rm) ,  and  Service 
Class  and  Programmatic  Scores  (Sk)  . 
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CHAPTER  5 

RESULTS  AND  CONCLUSIONS 


The  result  of  the  process  above  is  three  sets  of  weighted  scores  for  each 
candidate  baseline  specification.  These  sets  of  scores  are  organized  into  a 
set  of  graphs  to  better  illustrate  the  relationship  between  the  candidate 
baseline  specifications  from  the  three  views  of  the  evaluation  process.  The 
scores  and  graphs  are  documented  in  the  Evaluation  Results  Report. 

The  results  of  the  process  are  analyzed  to  formulate  and  justify  a 
recommendation  to  the  NGCR  Program  office  as  to  which  specif ication(s) ,  if 
any,  the  OSSWG  should  use  as  a  baseline  for  its  standardization  process. 
Discussions  concerning  which  recommendations  the  OSSWG  should  make  to  the 
program  office  is  discussed  at  an  OSSWG  once  the  scoring  is  complete.  This 
recommendation  is  arrived  at  by  discussing  the  three  sets  of  values  with  the 
OSSWG  membership.  The  final  decision  on  recommendations  is  made  by  the  OSSWG 
co-chairs.  The  final  recommendation  is  documented  in  the  Recommendation 
Report.  Other  issues  and  lessons  learned  are  included  in  the  After  Action 
Report . 
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APPENDIX  A 

EVALUATOR  TEMPLATE  FORM 


*  OSSWG  OS  Interfaces  Evaluation 

* 

*  Process  Management  Interface 

♦Service  Class: 9 
♦Evaluator  Name : 

♦Evaluator  ID: 

♦Candidate  ID: 


*9.1  Create  Process 

♦Score  (0  -  10)  : 

♦Confidence  Level  (H/M/L) : 

♦Rationale/References 

♦(text:  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦Comments 

♦(text;  don't  start  line  with  '♦'  or  exceed  100  characters  per  line) 


*9.2  Terminate  Process 

♦Score  (0  -  10)  : 

♦Confidence  Level  (H/M/L): 

♦Rationale/References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


*9.3  Start  Process 

♦Score  (0  -  10) : 

♦Confidence  Level  (H/M/L): 

♦Rat ionale /References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


A- 1 


NAVSWC  TR  90-248 


*9.4  Stop  Process 

♦Score  (0  -  10) : 

♦Confidence  Level  (H/L) : 

♦Rationale/References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 
♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


*9.5  Suspend  Process 

♦Score  (0  -  10) : 

♦Confidence  Level  (H/M/L) : 

♦Rationale/References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


*9.6  Resume  Process 

♦Score  (0  -  10) : 

♦Confidence  Level  (H/M/L) : 

♦Rationale/References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


*9.7  Delay  Process 

♦Score  (0  -  10) : 

♦Confidence  Level  (H/M/L): 

♦Rationale/References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦Comments 

♦(text;  don't  start  line  with  '♦'  or  exceed  100  characters  per  line) 


*9.8  Interprocess  Communication 
♦Score  (0  -  10) : 

♦Confidence  Level  (H/M/L) : 

♦Rationale/References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 
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*9.9  Examine  Process  Attributes 
*Score  (0  -  10) : 

Confidence  Level  (H/M/L)  : 

*Rationale/References 

*(text;  don't  start  line  with  or  exceed  100  characters  per  line) 


Comments 

*(text;  don't  start  line  with  or  exceed  100  characters  per  line) 


*9.10  Modify  Process  Attributes 
*Score  (0  -  10) : 

Confidence  Level  (F/M/L)  : 

*Rationale/References 

*(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


Comments 

*(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


*9.11  Examine  Process  Status 
*Score  (0  -  10) : 

Confidence  Level  (H/M/L): 

*Rat ionale/References 

*(text;  don’t  start  line  with  '*'  or  exceed  100  characters  per  line) 


Comments 

*(text;  don't  start  line  with  or  exceed  100  characters  per  line) 


*9.12  Process  Identification 
*Score  (0  -  10) : 

Confidence  Level  (H/M/L)  : 

*Rat ionale/References 

*(text;  don't  start  line  with  or  exceed  100  characters  per  line) 


Comments 

*(text;  don’t  start  line  with  or  exceed  100  characters  per  line) 


*(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


*9.13  Save/Restart  Process 

*Score  (0  -  10)  : 

Confidence  Level  (H/M/L): 

*Rat ionale/References 

*(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


A-3 


NAVSWC  TR  90-248 


♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


*9.14  Program  Management  Function 

♦Score  (0  -  10) : 

♦Confidence  Level  (H/M/L) : 

♦Rationale/References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦General  Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line) 


♦End  of  evaluation  form  (Do  not  delete  this  line!)* 
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Appendix  B:  WEIGHTS  TWO,  RAD  WEIGHTS  (Cont.) 
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